Well, in this part of the course, we're going to be moving into a new section, which is machine learning. There's a lot of material we're going to cover, but at the same time it's really the tip of the iceberg for all the material you can do in this field. Let me try to start by relating to something that we're familiar with, and then connect it to something new we're going to be doing, which is building models. So all of you have been running functions for a long time, right? So hey, maybe I'll write this code, and then our function take inputs, maybe in the form of parameters, and then they have some outputs, maybe I'm either printing something or I might return a value. And so for example, I can imagine that a function could be doing something like making some sort of prediction. Maybe my input is that I have some details about a house that's for sale, and then I might be predicting what it might sell for. And so when I have a function like this, that's an example of a model.  So the idea of machine learning is instead of having a human write these functions where I have a computer automatically generate these functions. And the way they're going to do that is they're going to learn by example. So we'll feed in a bunch of training data where we have a bunch of different houses that have sold for different amounts and have to have bedrooms and baths.  And then based on that function, we're going to be able to generate that function and then use that to make predictions on other data. And you can imagine why that might be useful for a lot of things. Maybe you're doing property assessments or maybe you're a realtor and you're trying to figure out how to price a house properly. So the example I've given here is an example of a regression model. And the regression model is trying to more broadly a type of supervised machine learning, which is one of the main three categories of machine learning. So I'm going to start brought now and talk about these three areas. And then we're going to be talking about regression in more detail and I'll actually explain what it is. So the three main areas of machine learning are reinforcement learning, which is basically a situation where you have to make a series of decisions. And you're trying to optimize some sort of reward. So you can imagine some sort of robot moving around in the world and picking up coins or something like that. We're not going to be doing that kind of work in this class. Instead, we're going to focus on two areas, which are supervised machine learning and unsupervised machine learning. And in both of these cases, we just have all our data upfront and we're trying to gain information about that. Some people will say there's a fourth category of machine learning called semi-supervised. We won't be talking about that here. Within supervised machine learning, there are two different things that we're going to learn this semester. One is regression and with regression, we're trying to predict a quantity and then with classification, we're going to try to predict a category. So in any of these cases where we're trying to predict something, that's known as a supervised problem. And the way it works is while the data we have has some labels on it. Usually there's some special column that's telling us a quantity like the price for a house or some sort of category. And then from that, we can try to predict that label in cases where the label is unknown. And unsupervised learning, there's no special label column that we're trying to predict. We're just trying to look for general patterns in the data. And so we might do a couple of things. One, we might try to cluster our data where we're placing rows into different groups. Or we might try to decompose our rows. We might notice that I might have these rows, which each have five numbers in them. But maybe every row is like a combination of kind of two component rows. And so there's some simplicity in there, even though there might be a lot of columns in our data. So I'm going to go through these four types of things that we're going to learn this semester and then just try to make it more concrete. So here I have a table. This is just a regular data frame. And so this is my index here. Here are my column names. Right now I have a Y column, which is my label. So that's going to be generally what I'm trying to predict. And then I have these different columns here that I guess I'm just running them X0 through X4. But usually those would have some sort of real name, right? Like before I saw that we have like the number of beds in a house. So this label that we're trying to predict is what we're going to try to do is look for a relationship between that and these other columns, which we're going to call features. And so in general, what will happen is that we have some examples, some rows where we have examples of both. And then there might be some other data where we only have the features, but we don't have the Y label. And so we want to try to predict what should go in here. And you can imagine why that might be. Maybe these are all different houses and some of them have already sold. So we know what they sold for. And then these have not gone on the market yet. So we're trying to predict, well, what would they sell for if they do go on the market? Right? So the problem here with regression, I just want to state again, is that we want to predict the quantity, the Y column in this case based on the features. Right? And by quantity, I mean this is like a number. So how are we going to do that? Well, we might actually break it down into three parts. First, we might select a subset of the data that we, for which we know the answer. And then we might leave some other data aside for which we also know the answer. And what we'll do is we'll run an algorithm that is able to infer what the relationship is between these features and these labels. Once I've done that, I might run my model on these other ones for which I also know the answer. Now, of course, I already know the real answer. And unless my model is perfect, it's probably going to give me somewhat different answers. And so why would I do this? Why would I want to make a prediction if I already know the answer? And the reason is that I can do this to evaluate my model or I might say test my model. So for example, if my model says that this row should have been 70 and it's actually 72, well, that's an error. Same thing here. 60 versus 59, that's an error. And I can try to quantify all of these errors and then give my model some sort of score. That's the testing phase. So after that, after I've learned my model up here and then I've kind of evaluated it on some known cases, then I might actually put it in production. Production means I'm using it for real things. And I'm trying to predict actual unknowns in the world. Like, for example, if I add a new house to the market, what might it sell for? And I can try to put these different values there. Other thing we might do, even beyond making predictions, is that I might look at that model and just try to learn things about the world. So I keep going back to the example where we're selling houses. I think it's interesting to just know, well, for each additional house or for each additional bedroom or bathroom I have in my house, how much does that increase the value of my house? And I can use it to make different decisions, right? Like maybe I want to do a housing remodel. Am I going to get more benefit by adding another bathroom or another bedroom? So we can just kind of learn things about the world and also make decisions in that way. Okay, so all this was regression, which was the first kind of supervised learning we're going to learn this semester. And when we're doing these regressions, the key thing that makes the regression is that we're trying to predict some quantity in our Y label. Now it's totally possible that our features might be a mix of both quantities and categories, right? So something like green, red, blue is a category. Something like shape is a category. A lot of things that are strings are categories. That's fine, right? The distinguishing characteristic of a regression is that the label column is quantitative. If I somehow am working on a problem where my Y is categorical, then this is no longer regression. It's a classification problem. But otherwise, all these other things I've been talking about where I kind of do training and testing and then I put in production. All of that is the same. We're just dealing with categories instead of quantities. Okay, so moving on, we saw the two kinds of supervised learning, which is both regression and classification. What about unsupervised learning? The main thing here, the main point is that there is no label column.  And then I can try to still learn some patterns about this, even though I'm not trying to predict anything. And so one of the things I might want to learn is, well, are there any sort of natural groupings of these rows? And so there are algorithms out there that will, let's say, put all these rows into three groups and I might assign them numbers like 0, 1, and 2. And then just to kind of draw what that looks like, well, all of these rows be lying together. Now, I really want to stress here that there is no data out there that tells me what the proper grouping is or even how many groups there are. So when I'm doing this, it's not exactly like there's one right answer. But that doesn't mean that all groupings or classifications are equal. I can measure within a group how similar those rows are to each other if I have some metric for that. And so then my goal is to have a grouping that kind of maximizes the similarity within each group. And there might be different groupings that are equally good, right? But as long as I'm kind of having a high similarity within groups, well, then I still learn something meaningful. And you can imagine lots of different reasons I might do this. Like maybe each of these things represent a different user for my web application. And if I can say, hey, well, there's these 10 different kinds of web users for my application, I could maybe run a different marketing campaign for each of these different groups. Okay, so clustering again is unsupervised. It's unsupervised because while there was no label column I'm trying to protect. The last kind of machine learning problem where I talk about this semester and which is probably the most complicated is called a decomposition. And decomposition is also unsupervised is again, right? There's no column I'm trying to predict here. And the idea with the decomposition is that I'm going to look through all these rows and see if there's any pattern. Are there kind of a couple archetype rows that really can be mixed together to create other things? So maybe what I see is that with some small error, most of these rows are just combinations of these three rows over here. And I would call these my component rows. So, you notice the columns are the same right between my original data and my component rows. And then to get this row here, like negative 11, negative 7, 3, 2020. What I would do is I would multiply this row by negative 11, and then add it to 21 times this row, and then add it to negative 8 times this row. All right, so I'm kind of taking a weighted average of these three rows to produce this this row. And if you actually crunch these numbers, you'd see that I would get something kind of similar to this, but there would be some error, right? It's not a perfect match, which is fine. I mean, the fewer components I have, well, then I can have a simpler model, but while there might be more error. So I have that here, and I have these numbers here. And what we'll generally do when I'm trying to mix these components to create a row is I'll put these numbers in another table down here. So this will be a table of all my weights or maybe my principal component scores. And so I'll put, you know, negative 11 here, 21 here, and then negative 8 here. And then for this next row down here, right, I'll do the same thing. I'll say negative 43 here, 12 here, and then the negative 6 here. And so since I'm doing this, I'm putting kind of these mixtures for every row down here down here. What that means is that if there's n rows over here, then there are also going to be n rows over here. If there are m columns over here, well, then there would be m columns over here. So basically what I can do is I can take this big table and I can reduce it to having some components here. I can have some weights here. It's useful for lots of things. One is just if I'm trying to save space on my storage system, right? I can have these things be smaller. But then it's also nice if I'm trying to do other phases of machine learning, like a classification or regression. It's kind of nice if I only have like three feature columns instead of the original five. That's going to help me in a number of cases. Okay, so that's a whirlwind tour of these four poms where I solve regression, classification. Those are both supervised because again, it's labeled, clustering, decomposition. There is no column we're trying to predict, so that's unlabeled. It's on supervised learning. And so for each of these four things, there's actually a ton of different algorithms out there. And in this semester, we really have time to learn like one algorithm for each of them. And so if I go to this website down here, this is the website for scikit-learn, which is the module we're going to be learning. And there's probably close to hundreds of different algorithms or different classes they have there. I put a small subset under here. And so I can see, well, here's all these different things they have for clustering. And we're going to just learn one of those, which is k-means clustering. Decomposition, all these different things I can do when I learn just one of them, which is PCA. It turns out that for a lot of algorithms, the classification and regression come in pairs. So for example, here I have like a decision tree classifier. Here I have a decision tree regressor. Here I have a k-neighbors classifier. Here I have a k-neighbors regressor. And that's why I didn't kind of split these out. I just put both of these under these two categories. And so we're going to learn two things here. We're going to learn logistic regression. And we're going to learn linear regression. And this is a little bit confusing because, well, this part's obvious, right? The linear regression is trying to be a regression here. This is the one that people get confused on because even though it says regression in the name, it is not a regression. It's actually a classification, right? So these are the four things we're going to be learning this semester. And logistic regression is one people always get confused on because, well, it's not actually a regression. And I think once we learn all these things, the very nice thing is that the interface to using the other ones is relatively simple. So, for example, once you know how to use a linear regression, you could very easily just replace the word linear regression with ridge.  Now, before you do that, you should probably learn about how ridge works and then think about which model is best for you. But at least in terms of the code, it's very simple to switch between different models within any of these four categories. So I want to talk a little bit, that was pretty high level. I want to talk a little bit about the foundations where you need to be learning this machine learning, both in terms of the code and then also the math. We're going to learn a few different modules. The main one is at scikit-learn. I was showing some documentation from scikit-learn. We're also going to learn numpy, which has, let's just deal with matrices. It turns out that numpy is really the foundation for pandas, right? All pandas data is actually stored in numpy. And now it will be a good time for us to actually see that. And then we're going to learn this thing called PyTorch. And PyTorch can do a couple of things for us. One is it can do calculus for us, which is pretty cool. Another thing it can let us do is it can actually let us run our code on GPUs, which are graphics processing units. Everything we've been running so far this semester has been running on CPUs, your central processing unit. And it turns out that GPUs that are originally built for graphics also happen to be really good at machine learning. And so a lot of things, if you're dealing with a lot of data or complex models, a GPU will be better at it. We're going to have to learn a little bit of math. I'm not assuming you have any math background besides what you might learn in high school. But let me give you an example of how math is going to come into play here for a regression problem. So we have this example again with all the houses and these characteristics. And then we have a function that predicts the price. How would we do that with matrices? Well, I might take all these numbers in the data frame and put them in this matrix here. And then for my function, I may have kind of just an algebraic expression, which is using matrices. So my x here is this matrix. C is a vector. B is just a number. And when I run this, well, I'm going to get this other vector out here, which actually has all the prices. So to understand what's going on here, we have to learn a little bit of linear algebra. This is not a regular multiplication. It's actually something called the dot product. And it looks like that. So I can take this x matrix here, dot product with this vector, and then add a number. And then that's how I'm going to get my results over here on the right-hand side and do one prediction. What's cool is that if I can do one row times this vector here and get one house value, and it's going to go through without having a loop even, right? The beauty of linear algebra and multiplying matrices together with the dot product is that I can do it in one step and I may get actually all of these numbers. The code for it is pretty simple. If I say data frame dot values, then x is actually going to be a numpy array. And if I want to, I can just say, well, I want the dot product of these two things and I want to add B and it just works. We're going to be talking about that in quite a bit more detail at some point before the end of the semester. One thing I want to note is that if you're reading other documentation, a lot of resources will use A instead of x, which I find confusing. I think that's not intuitive if you're kind of, you know, working all these scikit-learn modules because those will generally use x for data. And then even Stranger, right, will often have a C when we're in scikit-learn, but then they call that little x instead. So just be, you know, as we're learning linear algebra stuff, I just want to say up front and I'll say it again. Be aware that the variable names are a little bit wacky. So what is kind of the scope of linear algebra and what kind of things are we trying to solve? Well, one thing that we're not going to solve is something like this, y equals x squared. That is not linear, right? Anything quadratic or cubic or anything like that. I'm not linear. All we can do is multiply variables by numbers and then add things up, right? So this is an example of a linear equation, right? I just have, I have some different variables, right? And then I'm multiplying them by different numbers. One of the things where I notice is that the way we're going to be doing linear algebra in this course is we actually have very big matrices and a lot of variables and a lot of equations, right? So you can see here I actually have 50 variables. So I think the key takeaway is that more variables, more data, but simpler equations. What about calculus? So here I have that situation again with the house where I have some training data. So I have both my features and my label. It goes into an algorithm and that algorithm will basically spit out this formula for me that I can use to predict housing prices, right? Now it turns out that when I was doing this training, right? I had the original prices and the new price might be a little bit different, right? 140, this is 190, 240, 254, right? They're all a little bit different. And so what I can do is for this given equation I end up with, I can have some sort of function, total loss function that compares the correct answer with my model's answers, right? So I compare these two and I get one number out, right? That's how I like what my error is or how bad it is. And of course how bad it is really depends on, kind of, the numbers that are part of this equation down here. So the whole idea of this training thing with this algorithm is that I want to find out, well, what C can I do that is trying to make my error or my loss as small as possible, right? So we're trying to minimize something. And I don't expect you to take Calculus, but I know a lot of you have and in Calculus we're often trying to minimize or maximize things. And that's why it's trying to come into play a little bit here. The good news is that we don't have to understand Calculus. There's going to be modules that can do it for us such as this PyTorch thing that we're going to be learning. PyTorch is also going to help us be able to run our code on GPUs where we'll be able to do things like take two matrices, shove them over to GPU and then multiply them together and it'll just kind of, it almost feels like it's just magically going faster than it would if we're running on a CPU. And it doesn't take a lot of code to move it around. So PyTorch is going to be very powerful both in terms of Calculus and using GPUs. To conclude this video, I just want to talk about this difference between developers and users and then who we are. When I'm looking at this picture here, right, I'm feeding all this training data into a machine learning algorithm and then that's giving us a function we can use to make predictions. There are classes and I guess people in general who either develop new algorithms or write code and optimize code for existing algorithms and we'll just do a tiny bit of that but that's not our focus. We aren't trying to do machine learning research or come up with novel ideas. We aren't developers, instead we're going to be users of machine learning algorithms that come in Scikit Learn and so some of the questions that we're going to be interested in going forward for the rest of this class is well, which algorithm should we use in SK Learn? How should we pick it and I guess how should we configure it? A lot of these have different parameters. In terms of the data, how can we clean it up so it's trying to work well with the machine learning algorithm we chose and then finally when we actually use this thing we're going to get all these predictions and we can compare it back to the original and how do we want to score that? There's not necessarily one right way to evaluate how good or bad it is and so we want to get some experience with that as well. So that's a bit of a preview about what's coming up in the course and hopefully this is kind of a fun change of pace compared to what we've been doing. Well, in this video I'm going to be training a regression model from Scikit Learn to some COVID data in Wisconsin. So here I am on the Department of Health Services website data portal and I can search for COVID here and the data set I'm using is this one right here, the historical data by county. So there are about 70 counties in Wisconsin and what this data shows me is for each day. I wonder if I can draw the data browser here. It shows me for each date in each county all of these different stats. So for example, how many positive cases are there total, how many new cases are there, how about over the average of the last seven days and how many deaths are there and ultimately what we're going to try to predict and this based on this data is well how many deaths are there two weeks in the future based on looking at the stats for today. So I had downloaded this and I'm not going to do it again and then there was a fair bit of data cleanup that I had to do here and that's not the main point of this lecture so I have a notebook for it that does all the cleanup I'm just going to kind of quickly walk you through this some of the things I did here without spending too much time on it. One I pulled out just a few interesting columns. So for example, how many positive cases were there on average over the last seven days and then how many new deaths were there. We had a bunch of missing data so I just dropped anything with missing data and then I converted the date to an actual panda state time and in the process I dropped what hour it was I just want to get the date without having the hour that it was posted and then finally the documentation for this data set says that negative 999 really means that there's less than 5 5 in whatever that field is so it could be anywhere from 0 to 4 and so I think just for simplicity I just replaced that all with 0 we don't really know what that is you can imagine doing something smarter like maybe 2.5 seems more fair in some sense anyway so I get a data set that looks like this there's a lot of cases while there's no new cases and no new deaths maybe for some of these smaller counties and then of course for larger counties this definitely is not 0 and so then I go down a little bit more the other thing I want to do is I want to add a column which isn't just how many new deaths are there but how many new deaths are there two weeks in the future and so I had to do some trickery with time deltas to basically join the data and two weeks in the future and you can look at that if you're interested but in the end I get a data set that looks like this I know how many new deaths there are on this particular day and then this last field is how many there are after the specific one and I save all this to the Wisconsin COVID data set so that's what I'm going to be working with here and I'm just going to head over here and create a new notebook to analyze that so let me head here what I'm going to do is I'm going to import pandas for starters so I'm going to say import pandas as pd and maybe I'm also going to add import matplotlib.pyplot as plt and then maybe I'm just going to configure my matplotlib stuff first plotlib inline and then plt.rcparams let's just make the font size a little bit bigger great and so now I can actually get my data frame so I'm going to say data frame equals pd.read.csv and what do I want while that by all I produce from that other notebook which is wisconsin.covid.csv and let me just peek at that so that's all good and maybe just to make sure that all these things are not actually 0 I'm just going to say what does the data frame mean and so I can see on average in a given county in a given day on average one in four people have died and so what I'm going to be trying to do is I'm going to try to be predicting this field and I might either try this as a feature or this is a feature and this is going to be my label column that I'm actually trying to predict it's a quantity which is why I'm doing a regression instead of a classification often before I jump into trying to do a regression I'll do a scatterplot to just see if I can identify visually any patterns in the data and so I might do something like this I might say dataframe.plot.scatter and then I can say my x equals something and then my y equals something and so in both cases the thing I'm trying to predict my y is how many deaths are there going to be two weeks after this given date and then my x I guess for the first case I'll try the seven day average like that and so I can see a picture there and then the other thing I want to do is I want to say well if I look at how many deaths there were today what will that tell me about how many deaths there will be two weeks from now and I see a slightly different pattern there sometimes for these I like to say alpha equals 0.2 or something like that just give it a little bit of transparency when a lot of points are on top of each other and so I'll do that as well and so the other thing that I'm going to like to do as we're going forward is a regression model on both of these variables and so I want you to get a sense when we score that how that score corresponds to kind of the strength of the relationship in both of these so I'll start with this one how can we train a regression model that fits this to this and the first thing I have to do is I'm going to have to import it so I'm going to say from sklearn.linear model import linear regression this is the main regression model we're going to learn this semester so I'm going to do that and then if I want to I can create a new linear regression object just like that and so there's a few there's a few methods that we're going to want to learn with this so some important methods are going to be fit, predict we'll fit, score and predict and we're going to be running those things on our data right so let me come down here and what I can do when I'm fitting it or training it is I give it two pieces of information I give it my my x values and I'll give it my y values and for the y values that could be a series if I want so I could just pull out that one column for my x values it has to be a data frame and so let me just take a look at my data frame earlier what I'm going to do first is do a model corresponding to this plot right so I want to do it based on how do the deaths today predict the deaths in the future and so what I have to do is I have to pull out this one column these new deaths into a data frame so the way I'll do that is like this if I do this just new then what then I just did a series the way I can do a data frame is I can pass in a list and I can say some columns here right and so it kind of looks weird right that I'm putting a list inside of brackets but that's what's happening and in this case I'm really just interested in one thing which is just how many new deaths were there on this particular day this nice data frame which is going to work well for us and then for my y values I can just directly pull it out as a series and that will be death new two weeks from now right so these will be my x values and that must be a data frame and then this y values and this can be a series right and in general why is that why I'm trying to predict one thing but I might be making that prediction based on multiple columns so I'm going to head down here and I'm just going to copy these things I'm going to copy this right here and then I'm going to copy this right here just like that and I train it so fit means to train based on the data so I'm going to have a bunch of examples of my features and my labels right so I can do that and that's relatively uneventful and I would like to be able to visualize what it looks like what predictions is it making if I have new data and it turns out that since I'm doing the linear regression on just one column here it's actually going to be fairly easy to visualize because I can ask it for a given x value well what y value do you predict say l r dot predict and then I can have to pass in some sort of value here so I think I can pass in something like hey if there's 10 new deaths say how many do I predict two weeks from now and I have to pass this in as a list of lists because that was the shape of this up here and I can see that there's also this array thing which is an umpire array we're not going to worry about that too much for now the way I'll generally do these predictions is that I might create a data frame that will help me show a fit line that's one way we can represent the relationship here right if I drew a line on this that would tell me for my x variable what do I predict for my y and so I'm going to create a new fit data frame like this that data frame and then I may have a column which will be death new remember that's number of deaths today and for that I'm just going to try to put a bunch of different values maybe I'll draw from like 0 to 500 and let me just take a peek at that okay so that's my death new and then what I can do is I can actually pass that whole thing in to my prediction to to basically figure out what those y values are so I can say LR.predict and if I want I can put in my whole data frame like that and try to find out that column and again I get a bunch of these weird values in this numpy array but the great thing is is that I can say I want to just shove those things in a new column and I may call that this and then I might like to say something like predicted to emphasize that this is not real data it's just a prediction so I'm going to put this up here and now when I run this I can see that I have for a given number of deaths on today well how many do I predict there will be two weeks from now so if there's 500 deaths today I guess I'm predicting that in two weeks there's going to be 160 deaths for a particular county and so what can I do now is I can put df.plot.line and I can say x equals this thing and I can say y equals this other thing just like this and different models will give you different things but the linear regression right here that I'm using is trying to give me a straight line so I'm going to run that and there's some sort of straight line there and maybe I'll just make it red what I often like to do is after I have my regression line I like to compare it to my actual data and so remember I had these similar columns before and that was all my original data frame and for my original data frame I just want to draw a scatter of all those points and then at this point I'm not drawing some sort of prediction I'm actually drawing real data so I'm going to do that and I guess when I'm drawing it down here let me actually try to put that on the same one so I'm going to say ax equals ax and I may have the first one returned in ax and then maybe out here I'll have the alpha be 0.2 again and why did I put ax there there we go and so then I get these nice plots and I can see that's far too long maybe what I should have done is I should have gone to about 100 so maybe here I'll make my predictions over the range of about 100 and so I'm going to do that and I can see how that line fits okay well how well do I do I mean I might not try to intuit that and really what I'm looking at is for each of these points well I'd hope most of them are near the not line and I can see quite a bit of them are not and so the way we what is the variance and the thing we're trying to predict right so what is the variance and this thing of variance is kind of um a measure of how much the values typically differ from the average right so I can get my variance and the variance in this column is 1.35 and the idea is what we're going to do is we're going to say well how much do each of these points differ from the average of the points along the y-axis and then we're going to compare that to like well how much do they differ from that red line that I drew right if the line is good then then that variance relative to the red line might draw down a lot I'm sorry the variance relative to the red line might draw down a lot relative to the average which I guess I could draw is like a horizontal line and and so the way I can do that is I can come back here and just like I fit the data where did I fit the data I fit it right here I can check instead of fitting it I can score it right and what this is going to tell me is how much of the variance is explained by the model and that'll typically be a score between um 0 and 1 with 0 being the worst and then 1 being the best and in some kind of weird cases it can actually go negative and maybe we'll eventually talk about that but generally it's going to be between 0 and 1 and so I see this is not great okay let's try to do it for our other variable that we had as well right so I had this one here and just looking at the plots I might expect that this one does a little bit better than I think what was it like 9% so I'm going to copy these things down here and so I'm going to create a new a new linear regression object and then what then I'm going to fit it where was I fitting before, here was how I was fitting before instead of fitting to the deaths on today well let me try to fit to how many oh this I wanted to get the number of positive cases whereas let me go up and look at my data frame again I wanted to fit to the average number of cases over the last seven days so I'm going to do that now I'm going to train my model by this new model based on this other variable how can I use this to predict how many deaths there will be in two weeks and maybe right away I'm just going to score it as well so maybe I'll just copy this and then score it and I can see it's doing quite a bit better instead of explaining 9% of the variance I'm explaining 20% of the variance if I was explaining 100% of the variance well that would be very remarkable because every point would be exactly on that line let me try to plot it just like I did before so I'm going to copy this here and so I'm going to have this and then remember that for this fit data frame I had to generate that by using a range of values so I'm going to paste this here and so what am I going to do again I'm going to generate values from 0 to 100 and that is telling me what was the positive seven day average cases and then I'm going to get a prediction based on that and I can plot a line like that I'm going to do that and I can see well something a little bit weird I guess I didn't change everything I see the x-axis is still how many deaths there were on a day as opposed to what I'm actually training on which was the seven day average so let me fix that and run that again and so that's a little bit better I can see if the line doesn't extend very far that makes sense previously when I was saying well how many deaths were there on a given day the number was relatively small compared to the number of positive COVID cases and so maybe I'll I'll redo this instead of going from 0 to 100 maybe I'll go from 0 to like 900 and so I run that and I can see this is a better fit right now I'm explaining 20% of the variance instead of just 9 so I did something bad here and I want to talk about that in the next video but I just want to leave you with a thought let's say I'm teaching something in class and I work out an example and so maybe everybody sees that example and they try to learn from it if I put exactly that same answer exactly that same example on an exam what does it mean that somebody does well on that exam I think there's two possibilities one is that maybe the person genuinely learns something from that example and then even though they're seeing the same example again they understand what's going on the other possibility is that maybe somebody just memorized the answer to the example and then when they see it on the exam well they just repeat it and that would be not so great and so the same thing happens here when I'm fitting I'm really giving my linear linear regression model some examples and and then when I'm scoring it well I'm actually using those same examples and so if this score were good and I guess here it's not great but if it were good then I wouldn't really know well did the model effectively memorize the answers or was it overfitting overfitting is what we mean when it basically memorizes it so next time I'm going to talk about how we can actually deal with that problem and did a better sense of whether of whether it's doing a good job so last time I talked about this issue of basically evaluating our model on the same data we fit it to right the model can effectively memorize the answer and look like it is doing well even if it didn't really learn anything and so the way we'll deal with this problem is we'll use something an SK learn called a train test split and this is a general strategy but this specific method will make it easy for us and what we'll do is we'll take our original data frame and give us back two data frames we'll train our model to one data frame and then we'll evaluate it on the other one on the data and the model has not seen before so we'll do a test of whether it's doing well and so I can see that there's a number of things I can pass into this one is I can say what ratio of my data I like to go into my test set I can say other things like try to stratify the data let's say I was dealing with something categorical I might want to make sure that I have a similar number of categories on both sides I'm not trying to do that here but I will specify a train test set I'm going to say from sklearn import it's sklearn.modelslection import train test split and then I'm going to say train test split and I'm going to try to put on my data frame that I have originally and basically it's returning something called an array with two data frames in it and so the way I can capture this is I can say I have my train data frame and my test data frame is in order and so let me take a look at these here's my train data frame and then here is my test data frame and you can see that it kind of shuffled things around if I look at the index and I'm dealing with different data right so this is the data frame that my model is going to learn from and then this other one is the one I'm going to use to actually make sure it learns something and so I can see right now if I look at this there's 6300 down here and then 19,000 up here and that's because the default split is 0.25 so I could do something like this if I want and then that would make this first one bigger and the next one smaller so I'm going to do that and then let me just see what I did wrong here I think I actually have to say test size so I'm going to say test size equals that and then this is going to be a little bit bigger more data to test on and then a little bit less data to train on and so there's some trade offs there anyway now what I'm going to do is I'm going to head back to this example that I had earlier where I was doing my training and the idea is instead of fitting and scoring on the same data I'm going to fit to my training data and then score on my test data so I'm just going to do it again from scratch and I'm going to do a review and I'm going to say LR equals linear regression and there we go and then I'm going to say LR.fit and I have my train data frame and then I have a list of columns here and then here I may have my Y column so that's the general strategy and then what I want to do is I want to score it I'm going to give it a Y column and an X column again, sorry I have to say train DF here I'm going to give it that same information but now I'm going to be doing the test data frame I'm going to say test data frame and then same thing down here test data frame and then what I'm going to do is for this I'm going to pass in a list of columns like before and what was I doing last time I'm going to look at this positive 7 day average and so I'm going to pass that in just like that and the same thing here and then over here I can just look at that thing that I'm trying to predict my Y which was this thing I'm going to pass that in and then the same here I'm going to do that and I see that okay my score is 0.16 whereas what would happen if I wanted to run it on my original data frame like that, training data frame and here it does better on the data I trained it on so this suggests that there is some overfitting here right the model does better the data I learned from instead of some new data it hasn't seen before and so that of course is a concern and so there's different things I can try to do to I'll avoid that and we'll eventually talk about some of those things this semester today I'm going to be talking more about how we can determine whether or not a model is doing a good job or not how can we interpret different scores and make sure that we aren't getting a good score just by chance and so I'm going to be picking up last time with the model that's trying to predict COVID deaths two weeks out based on the current number of cases and so let me just quickly review what I did last time we were doing a train test split on our data frame and our data frame looked like this right here and it was per county and then there's every day in the year there then I created this linear regression model which I imported from sklearn.linear linear model and then I did two things with it I fit it to my training data and then I scored it on my testing data and these were the two pieces my training data and testing data came from here and so when I was doing this what I basically did is I put in my y values which is what I'm ultimately trying to predict and then here I put in my x values or my features which are things I know right now so for example right now I know the 7-day rolling average of positive test cases and then two weeks out I'm trying to predict well how many deaths will there be this can be a single series which is why we just put a string in the brackets after the data frame here we actually have to pass on a full data frame because in general we might have multiple features and when I pass on a list to the brackets after a data frame well I get a smaller data frame that's why I have the double brackets there so anyway so I have this 0.2 and we know that this score will somewhere be somewhere between 0 and 1 so it's a little bit hard to say how good this score is right maybe you always get something like 0.2 by chance do we know so that's one of the things I want to talk about today and then the other thing that you might notice if I rerun this a few times oh now I'm at 27% 32% 24% 28, 26 so you can see that based on how I do this train test split I can do very different numbers and so that doesn't give us a lot of assurance so how can we get some more stable numbers I'm out of the system I'll just give you a hint of what the problem is if I look at trainDF and I look at this column here which is the thing we're trying to predict let me do that so I have all those numbers there let me look at the variance of that column variance is just kind of a measure of how different values are from the average and then I'm also going to do that same thing for the test and so when I do this for the test I see that actually they have quite different variances and if I run this again while I just randomly shake out differently now the test data actually has a higher variance than the training data and we're going to eventually look at how this scoring function works but it turns out that it's very much based on this variance which is why we have such a noisy measure so let me head over to the slides and try to give a preview of the things we're going to talk about today we're going to be learning four new functions related to model evaluation so first we're going to learn these two functions here which will let us score our models second if our model is trying to meet mediocre like mine is I mean 0.2 is not great how can we know if it's not just chance and then we're going to be using something called the permutation test score for that and then finally to get a less noisy measure we're going to be doing something called cross value scoring let me start here with these two metrics here I have this R2 score and mean absolute error and we'll talk about how those work so if I go back to my slides or go back to my notebook right here I'm doing this scoring here I can also up here I can say from sklearn.metrics import there's a couple things I want to do I want to do the R2 score and then the mean absolute error score and so the way all these metrics work is kind of like this I'll call the metrics function and then I'll say something like let me actually just check this here quick I'll say something like I had to make sure the order is different between my true and my predicted I'll say what are the true values and then my predicted values and so for example up here what are the true values and then my predicted values well how do I figure out my predicted values I can just say well model please predict for me what these y values should be based on these x values that I'm going to give you so I'm going to do that and so here I'm putting in x values and this is going to return back to me y values and this weird array thing that we'll eventually talk about but I could take this and I could put this right here and so then I could use lots of different metric functions here I could for example use the R2 score right here and guess what it turns out that this score here that's associated with the model is just defaulting to use our R2 score there's lots of different metrics I could have used instead but this one is the default one so let's talk a little bit about let's talk a little bit about how this is computed so the idea of it is that this is the thing I'm trying to predict right so this is my y column as I'm going to say y is here and I can just peek at that what I want to do is I want to look at basically the squared residuals of this column relative to the mean and so what does that mean so I can take this and I can subtract off the mean of this and then if I want to I can subtract all of that so this is really a measure of kind of how bad the system is if I add all of these things off so this is really like the variance of the system except I'm summing instead of averaging right so this is my original kind of total error variance of the system I might think of it as I'm just adding off so I'm trying to have a sum of squares and then what I want to think about is what if instead of measuring the distance from each each value to the mean what if I measure the distance to my actual prediction right so it'll be very similar here right I can say you know what what is kind of left over if I subtract off my predictions how do I get my predictions well that's this right here so I'm going to grab this piece and I'm going to have my predictions here and then let me take a look at that so the way I really think about it is that originally this was how much variance I had in the system and this is how much I have after I do predictions rather than just subtracting off the mean and so what I could do is well I could say I could say well how much is remaining right so I could say left over divided by the total and why is that let me just put that here I can say that left over divided by the total and I'm like okay why left 79% of the variance on the table basically which means that I took away 20% and you can see that that's exactly what this is up here right so really this is the math that people are really kind of used to evaluate how good a regression is typically we can do that more simply with R2 score and even more simply that's the default of I just say linear regression dot score let me give you an example of another metric people might use so maybe I want to just get well what is the average error and so in that case I would go back to this piece right so this is all of the errors and if I want to get the average I should probably just think about the absolute error and then I could take the mean of that right so this would be the average absolute error and it turns out that that is just rather than me collecting that myself I could just grab this mean absolute error here there's lots of different metrics in here and I think if I hit shift tab well maybe it's just regular tab you can see all the different metrics that come here and most of them I have never used so I paste this here and I see well that's kind of strange it should be the same it should be the same number why is that not the same number because I wanted to get the error my prediction is not relative to the mean and so I'm sorry this was the thing that I wanted to this is the thing I had wanted to grab so I wanted to say here are all my errors take the absolute value of them some errors are positive or negative I just want to have the absolute value then I have the average and why is that invalid syntax that usually means I have a mismatch in terms of my parentheses I see so this one has matched up there so I don't know why I grabbed that squared so I can see that this is all my errors and then I'm taking the absolute and then taking the average and now thankfully I want to actually get the same thing that I have down here and so again this is just a shortcut for this kind of more complicated map but it's another metric in terms of how these metrics work this one is kind of counting all errors more equally an error that's twice as big is just twice as bad because this one up here where I'm doing the where is it right here where I'm doing the R squared score because that one is squaring my errors this will tend to make it look worse if I have a few errors that are really big as opposed to many small errors okay so those were a couple metrics which was one of the things we wanted to answer on the slide so I'm just going to head back here we talked about R squared we talked about an absolute squared error and that's our different ways we can measure our errors the next thing I want to talk about is if our score is kind of mediocre like 0.2 out of 1 how do I know that that's actually go ahead and it's not just chance and the answer is what we'll do is we'll take our original data here and we'll score it with our model, we'll train a model of the x's and the y's and we'll score it as 0.8 which is not that good what I'll do is I'll shuffle around the data so I know that there's no relationship between the y's and the x's so what I'll do is I'll take this y column and just randomly shuffle it and the word for that is permutate it as I get this permutated version of this column you can see that for example 5 used to be the first number and now 5 is down here so I'll shuffle that thing and then I train the model on it trying to look for the relationship between the y's and the x's and there should be no relationship obviously because I just shuffled everything around but I can try to train a model and get a score on it and so if I see that when I'm basically training a model on garbage data if I get a better score than I did originally well that probably means that I probably didn't have any sort of meaningful model and that's the rough idea in actual implementation what this function here is going to do for us is it's going to shuffle around the data like this and it's going to get a score and then it's going to shuffle it again and it's going to get a score and it's going to get something like a hundred or a thousand different scores based on these shuffled data and then based on that we can estimate and really see this score over here how good or does it feel like this could fit in with the garbage data and based on that we can basically say well hey do I trust this model or not so I'm going to head over here back to the notebook and so maybe I'm just going to make some notes in here just so it's clear what we're doing so this part was about metrics and then this part is going to be about permutation testing so let me I actually already imported it which is great so all these things are kind of related to model evaluation and they're under this thing called model selection which is what we'll often do is have a few different models and we're trying to have different tools to say well what is the model that we think is best that we recommend to people so I see that I have this permutation test score and I'm going to paste this right here and I can see that I need three things I have to have my model which is just LR I have to have my x values and then finally my y values and so the way I'm going to be doing this here is since since here I have this test df and and for both my x and my y I'm just going to grab these right so this was my test right here based on the 7-day average I want to predict this right here and so I'm going to grab this and it turns out that this is going to return a tuple of length 3 so I'm just going to run that that'll take a moment and the three things in that tuple are going to be the score of my module model originally it will be maybe I'll just call it the garbage score since I'm permuting the data I'm not expecting to have any pattern there's going to be a bunch of those and then there's going to be something called a p-value and what the p-value is telling me is well what is the probability that a score this good would be generated by a system that's generating all these garbage scores right and so if this is really small then I can see well this is actually much better than my garbage scores and so I actually have a significant result so since these are the three things that it's returning right I know that t is a tuple I just put that here and it will automatically unpack those things for me so I'll take a look at this and then we get a score for my model and then I can have my garbage scores here I see there's a whole bunch of them if I want to I can put those in a series and and then I can do a histogram of those and I can see that there are actually around zero or less right all of these and this was around 0.09 which is actually pretty far over here so it seems like we're pretty far away from these garbage scores and therefore this p-value is going to be pretty small it seems like whatever process I'm using to get all these garbage scores is not likely to have a score this good so I will take this as a meaningful result let me head back here for another idea so how do we deal with the noise we saw that when I kept doing it a bunch of times I was getting different scores and for that we're going to use something called a cross-validation cross-validation score and the way it will work is I'm going to split my data instead of just having these train and test I'm going to split into four pieces and then each of those four pieces are going to take turns being the test data so maybe first I'll train my data on these rows and then I'll test it on this and then my model will get let's say a score of 0.2 and then I'll take a different chunk of the data and each of these are called a fold of the data by the way and so I'll train on those first, second and fourth pieces so I get a model and then I evaluate on that test data set and let's say this time I get a little bit luckier and it's 0.3 do it again 0.1 again 0.2 and then I can take the average of these and that would be a more stable measurement of how well my model does it's not as vulnerable to what happens to go on the test or training data set because all the data at some point isn't the test data or the training data so I'm going to head over here and do this so this is called cross-validation and let me call this thing so cross-validation score and what do I have to pass in here I have to pass in my estimator and then I have to pass in my x values and then my y values right so let me grab those things right so it's actually I guess identical to this right here so I'm going to grab my model my x values and my y values and this time I just want to do it on all my data so can I say data frame and then data frame and actually really what is kind of the best practice is just the training data and so I'm going to do that and then I get all of these scores back and the reason is that there were 5 folds by default so I can say in this picture right there is 4 here I see by default there were 5 that's why I got 5 scores and that'd be fine I got all these 10 scores back and these kind of look like those numbers we were seeing earlier like 0.27, 0.17 another 0.27 is 0.304 right if I head back here and I just kind of run this thing a few times those are the kinds of numbers I'm getting out of it as I randomly split my training my test so why is this useful well I can have my scores here and I can say a couple things I can say scores.mean and so I can see well on average my r2 r2 score is going to be 0.21 but I could also get some sense of the variance and that'll tell me how sensitive I am to what data happens to end up in the test or training data set and that probably depends on how much I have some outliers and how much outliers define what happens with the scoring ok so this will be the way we'll generally do it and so one last thing right when I was showing this picture here right I kind of said well hey I have all my data and then I just split it up into training tests why didn't I use all my data here and the reason is that is that even though that would have been fine to do in this example what you're often going to be doing is you're going to be trying to do a few different models and what you'll want to do is you'll want to do cross validation on each of the models and then you'll see well what one has the best score on average and you say well that one's the winner that's the one we're going to use in the future and so there's this risk when you're doing that let's say I evaluate 20 models and pick the best one the best one probably did a little bit better than it showed right if I do 20 models some will just by luck do better and some will do worse and so even though that's the right process to pick the best model I shouldn't go brag about this cross validation score because it wasn't like I was just doing one model as in here I was doing many models so what I would do is I'd look at this cross validation score across each of my models pick the best one and then finally when I go back and actually do my real test data which is still hanging out here and that's what I would report is the kind of accuracy of my favorite chosen model well in this video we're going to be learning about scikit learn pipelines we've been learning how to use linear regression as a model and often our model will have to transform the data in some sort of way before we can actually analyze it and so what we'll end up with for our models are these pipelines where we do a series of transformations and then at the end we'll use an estimator is what they call it in scikit learn to actually make predictions and so for this example I might be using a slightly more complicated data set in Chicago which is right on Lake Michigan they have all these sensors on different beaches which are measuring things about the waves and sort of looking at this data set of all these measurements so I can see well here's an Ohio beach here's 63rd street beach and I know all these things about these sensors like how warm is the water turbidity and other things like that and the why that we're going to try to predict is well how big are the waves on this beach and we'll be using things like like wave period to predict it or maybe by looking directly at what beach we're on and so there's some garbage data in here so I've cleaned that up so this is a picture of all the data where I have the wave period on the x axis and then on the y axis I have wave height then I also try to break it down and look beach by beach so here I'm trying to pull out all the beaches as a sorted list that's unique from the beach name and then what am I doing down here I'm drawing each beach separately so I'm creating some subplots and I'm looping over those and I'm basically plotting each beach right I'm doing some filtering and in pandas I'm plotting each beach in a different AX region and so I look at this and so there's a couple observations right away often before I do modeling I like to just do a lot of scatter plots and try to get an intuition for what's going on one is that I do see that what beach we're on is an important variable some of these beaches have different patterns in terms of the waves the other thing I see is that that the relationship between wave period and wave height is not linear it's not as if the bigger the wave period the bigger the wave height or vice versa what I see is often there's kind of like this hump in the middle right so kind of if we want to get the biggest waves it needs a wave period that's somewhere in the middle so I'm going to be making four different models here that try to analyze this data and they're just going to be using different variables in different ways first will be something very similar to what we've done before I'll try to predict the wave height based on wave period just using a simple linear regression and we'll see what performance we have there next we're going to learn how we can do a polynomial so that means if I'm drawing a line on here well the line doesn't have to be straight anymore finally I'm going to look at well not finally but next I'm going to look at what beach if the only thing I know is what beach I'm on how well can I predict and then finally I'm going to look at the combination of both beach and wave period with wave period being treated as a polynomial then see how much I can predict there so in terms of my imports I have some things that we've done before we've the first thing we learned was how to do a linear regression we would fit our linear regression to training data set and then we would evaluate it on a test data set at the very end along the way we might also do cross validation within just the training data set so these things are old and then these things down here are new right so in order to be able to deal with this data and I want in a polynomial way I might have to transform it using this polynomial features thing if I want to deal with categorical data like well what beach am I on that's not a number right that's a category I have to use this thing called one hot encoder and and if I want to use both of those I have to use this other thing make column transformer to combine them and in total what I'm going to do is I'm going to have this pipeline and in the end of the pipeline I'm still going to be doing just a simple linear regression but I'm going to be making all these transformations to the data before it gets to that and that's why maybe I'll do these more complicated models so let's start down here and I may call I'm starting to number these four models one two three four just following the numbering up here it's I may have model one is going to be just a simple linear regression like that and and then what I could do is I could say M1 well actually I could do different things I could say M1 not fit and I would fit that to my x values and then my y values here this is uppercase because it's a full data frame and then this is lowercase because it's just a single series and I guess before I have to do that I actually have to separate out my train data frame from my test data frame I'm going to say train test split df and maybe I'll just look at the length of both of these and so about 25% is going to testing which I'll be fine with that and so then down here I have to have my train df and then here I can put a list of features and then over here I may have my train df again and then I'm just going to have my y column and so my y column the thing I'm trying to predict is just well the wave height like that and then my list of features well first off I've put a list here and that's why I get these weird double brackets I'm just going to start with wave period just like that and I probably have to capitalize to match what is in my data frame from earlier if I check back yep it looks like all these things are capitalizing and so I can do that and now this model has learned the pattern and so then what I could do is I could say m1.predict based on my testing data so I could say test df only x values and then it's going to predict what these y values are for me and so let me why did I call it m2 m1 and so I get all these predictions and then what I could do is I could compare those predictions to what I actually have for that wave period right I could compare these numbers these numbers down here in any glance I see it's not very close the easier way for me to do that comparison and actually get a score will be I call this scoring right so the score function will automatically call that predict on this and then it will compare that to my test df of wave height and it will tell me what percentage of the variance I'm explaining there and I see right now it's terrible I'm not really explaining anything if I just always predicted the average wave height that would be better than what I'm doing right now now of course some of this is lock in terms of like how I did my train test split and so the more reliable way to get a measure here let me just print that would be to do my cross validation so I could say cross validation score and then here I have to give it three things I have to say well what is my model my model is m1 what are my x values and what are my y values I'm going to do this up here and when I'm doing the cross validation I usually do that all within the training data and I just try to hold back the test data at the very end of my project to have a final analysis well I see I actually get a variety of scores and we can see that yeah it is a matter of luck sometimes I do worse than zero most of the time not I can say how many pieces I want to break my data into remember that I cycle through each piece or fold and it has its turn being the test data set so I can get more numbers here maybe I should say scores equals that and I could say scores dot mean and this is probably a better indication of how well my model is doing I'm explaining one tenth of one percent of the pattern which is not too surprising all I'm using is the wave period and I'm trying to fit a straight line to this and I can see well yeah no surprise I can't fit a straight line to it okay if I wanted to let me just copy this for a moment I'm going to delete it shortly if I wanted to do more columns that would be an easy thing to do so for example I'm going to include let's say like the water temperature I could easily add that as another column right it's very simple to have multiple x values right so I can do that here and then down here as well and then maybe I can get a slightly different score still not great right I'm just going to delete this here right but it's very easy to add these different things and why is that well when I do this and I put that list here I'm just going to create a name of x columns that I want to use for my predictions okay so that's how we can add things let's actually think about how we're going to do this now if I want to have a polynomial so what we'll do is let me actually delete all of this I'm just going to create a demo here which is going to be a copy of my train df and in my train df I may have just wave period let me take a look at this data frame what we're going to be doing if I want to have a quadratic fit I want to just add some columns down here that are going to contain that squared data so I could do things like this I could say demo I could say period squared equals demo of wave period or I could have one that's cubed like this and why is that unhappy because I'm doing a comparison on an assignment so it's assumed that's already exists and so if I do that and you know what I want to do I actually want to copy this so that it doesn't complain I can see what I'll do is that this column 2 squared is 2 2 cubed is 8 3 squared is 9 3 cubed is 27 and so what I can actually do is I can do a linear regression across these things even though it's technically a linear regression it's trying to act like a quadratic or cubic regression because I'm just treating these as regular columns and I can put weights to how important these things are and so how can I do this so here I was just manually adding these things if I want to I can use this polynomial features thing and so down here I may say poly equals polynomial features and I can say poly.fit transform and what I want to do I want to transform my data right here this was what I had before so I'm going to run this thing and I see that I get all these different columns and if I wanted to let me capture those and data right this is one of those numpy arrays which we'll eventually learn about until we learn about it I just want to put it in a pandas data frame so I can better see what's happening so I'm going to say pandas.dataframe and then I'm going to put my data there and then I want to figure out what the column names are on those and it turns out that this thing will tell me that as well so I can say I'm sorry poly will tell me that so I can say poly.fit feature names just like that and I actually have to say columns equals that and now I can see well I have x and then x squared and I actually have to tell what my original name was for that to work I actually have to have a list of that excuse me and as I can see okay well I have period and I have period squared and if I wanted to appear I could say things like well I want to have it be fourth degree right so I could have period, period squared, period cubed, period to the fourth you can see it also is giving me period to the zero which is just a column of ones right and so what I'll often do is I'll disable that it's called the bias column so I'll say I don't want that thing and so now I actually have something that's very similar to what I did above manually and I could do that with my data right I could if I wanted to I could do this transformation with my training data and then my test data and then that's what I would do all my modeling on now I think to keep things simpler what we'll want to do is we'll want to automatically transform and then immediately apply the linear regression and so it turns out that pipelines which I also imported up here make that really easy to do so I can create a pipeline down here like this maybe I'll just draw this this is going to be my second model pipeline and then we have to pass in a list here and the way the list works is that I will have like transformers um and I might have like one or more of those and then at the end I'm going to have an estimator right so all these things are going to be modifying my data in some way maybe adding more columns and then at the very end I'm going to actually do my real model and my real model is just a linear regression like that and then here well what are my transformers well I'm just going to do polynomial features like this and and I think degree of 2 will be fine for us for now and so I have these two things and then the other detail about this pipeline is that we have to name each of our stages of the pipeline and the way that it wants us to do that is it wants us to put these things in parentheses to create a tuple and then put a name as the first part of the tuple so I'm going to call this transformer poly because it's a polynomial transformer and then this thing I don't know I'll just call that LR for short right it doesn't really matter alright so let me just take a look here M2 it kind of shows me all these details of it but now M2 is just a model right and it's a lot like a simple linear regression I can do it in all the same ways so for example if I head back here all of the stuff I did before you know I just created M1 as my model and I did all this stuff on it I can do all those same things down here right so if I run down here and I just say M2 dot fit and maybe I'll just delete this for simplicity everything is the same and I can see well maybe I'm doing slightly better now right so what was my score before that's exactly the same I was expected to do slightly better and why wasn't I doing better probably because I was evaluating my same model that's the curse of the copy paste so before let me just see this I was explaining 0.1% well I guess closer to 0.2% of the variants and now I'm explaining more like 4.6% so I can see that model 2 is a huge improvement on model 1 I think I really did there I'm still doing a linear regression but I'm just giving it a little bit more information to work with by giving it these extra columns I have a column that says something like wave period squared and going way back here the intuition is right as I'm fitting a line here and that line doesn't have to be straight anymore it can be a quadratic line that kind of curves okay so we have two of these models we have this one which was terrible right it told me almost nothing sometimes this one depending on your train test data really lead you farther away from the truth and if you just always get the average this one is actually doing somewhat well right it's explaining almost 5% of the variants let's try the beach and see how we can do that so that will be my third model and so if I come down here I think for this one I'm just going to go back to here and try this as a first attempt maybe I should have some comments here so this is poly on period and then what was this one up here so model 1 this was just linear on period and then I'm going to try doing linear on beach is what I'm trying to do right now and so if I have this model I'm going to call this model 3 and I'm going to delete this again just to keep it clean I'm going to do this down here the main thing I want to do is instead of having it be wave period I want it to be the beach name if I have it all the way up here I just see well that's beach name ok so I'm going to head down here and I want to protect the wave height based on beach name right so just like that and then this is also beach name down here and this is going to complain to me and what does it complain about it says could not convert to a float Ohio street name street beach is not to not be converted to a float and that makes a lot of sense right so this thing here if I take a look at it is that's categorical data and it turns out the linear regression will actually learn why but it wants everything to be numeric and so that's a problem and so how do I how do I deal with this I mean maybe one idea that students sometimes come up with is that I could encode these I could say like one means Ohio beach and then maybe two means Calumet beach and then maybe three means something like Montrose Beach and the problem is is that if I put these numbers like this um the linear regression model is going to assume they're meaningful and so what that means is that if the model learns something about Ohio Beach and then it learns something about Montrose Beach it mistakenly thinks it knows something about Calumet Beach just when I think of that somehow the average of these other two and of course that's not true right I mean just I kind of arbitrarily put these numbers here there's no reason to believe that this beach has characteristics that are kind of average of other two so that's not going to work right I can't encode it that way the idea that we're going to use instead is called one-hot encoding and one-hot encoding looks like this one-hot encoder it's going to be like that and I can say one-hot encoder.fit transform I'm going to fit transform this data right here and I get this weird thing that's well whatever is that that's like a it says it's like a sparse matrix we're going to eventually learn more about that but I can convert it to this thing which is um is a numpy array and then that thing I can actually put into maybe I mean simplify this a bit that thing I can actually put into a data frame or something like a pandas.dataframe just like this and then just like before I want to figure out what these columns are and so just like with the polynomial transformer I can say get feature names same deal here right I can say one-hot.get feature names like that I have to tell what was I originally operating with and I guess I was like the beach and why is that unhappy so the shape of the values past is this and the c's imply something like that I wonder what is drawing out right there I think my problem is that I need to say like columns equals that um okay and so how is this working so the first one my first one was Ohio Beach and you see that I have actually a column for each different beach and what we'll do is we'll set within that we'll set it to one if it's an Ohio Beach and then it'll be zero and all others um if I go a few down I see that I have a Montrose Beach in position four right and so that case I'll put a one under Montrose Beach and then a zero is under others and that's why we call it one-hot right so I guess the place where it's hot is where we have a one and then everywhere else it's a zero so if I have this even though I started with some um categorical data I can end up with something where um where I have a bunch of numbers and so if I clean this up here let me throw back to this I may actually delete all of this because this was kind of a dead end right what will be a closer inspiration is the pipeline I used before with um with the polynomial features like polynomial features is a transformer um so also is one-hot and toting right so I'm going to tweak this we're on model three now and this is one-hot and here I can just blow all this away and I can say one-hot and toter just like that and then I can do my linear regression down here and so then I can say um down here I can say uh beach name right so beach name and then uh actually I don't even need this right I'm kind of this will automatically do the fitting for me and tell me how well it's doing um I wonder why this is complaining up here I didn't nervous that's read anyway it'll probably tell me shortly um I can say beach name here and then let me try that and then sure enough it's invalid syntax um because I didn't have a match there and so this does even better than before right so I guess just looking at the beach name is even better than knowing what the wave period is right so my model is getting better and better right first I started with just a linear model on uh on the wave period and then that's terrible right I'm not even explaining one percent then I said well let's do a polynomial fit on uh on the wave period and then I'm explaining almost um almost five percent and uh four and a half about four and a half percent then I just look only at the beach and I get five point five percent and so the um kind of natural next thing to do is well um let's do the regression on both beach and um and also a polynomial of the um wave period and maybe I can do even better right maybe the both of these things can give me some information and so here's where I get to have a challenge right because I want to do one hot encoding on the beach name and I want to do polynomial features on my other column right so I have to have some sort of wave combining these things and it turns out that that's that last piece that I imported up here the make column transformer is going to let me stitch together multiple transformers each of which applies to a different column so I'm going to come down here and um and I'm going to copy this for my inspiration for um for kind of model four so I'm going to have model four now and what am I going to do so this is what has to change right so I have to have something here that can capture both columns right because what I'm going to do down here is I'm going to pass in both beach name and wave period like that in an effort to it to predict my wave height and so how do I do that well I'm going to call that thing that I imported which was make column transformer and I'm calling that thing and when I'm calling that what I'll do is I'll have just a series of transformers and so I'll have like transformer one and then transformer two and if I wanted to I could have more in this case um each of these things each of these transformers is going to be a tuple and the tuple will be the transformer and then it will be um a list of the columns that it applies to right so both are going to be like that right I have these um tuples and maybe I'll just try to put a new line there so I'm going to have these two transformers and then a list of columns and so I think what I want to do is I want to do um one-hot encoding of the beach name so I'm going to copy my one-hot encoder right here and then what column does that apply to well that applies to the beach name column and then my other one was polynomial features which I should just copy from up above that was this thing right here polynomial features is my other encoder and what does that apply to that only applies to wave period okay so I have this um this model four now and uh and then I can well try running it and now I see I'm getting up to 9.5% of varying it's explained which is actually well I guess I was wishing I wish I was explaining 100% but I can see by uh kind of considering both these factors I'm explaining quite a bit more um than I would have otherwise um so one last thing I want to do before I wrap up this video is I want to talk about why we have these names here and um I can use this um we can uh use pipelines like a dictionary right so um so for example that means I can put both here and what will that give me that will give me this column transformer that I created right here and so if I wanted to I could um kind of peek at what this thing is doing as a way to debug my model I could say fit transform and then what I could do here is I could say well this is the data I'm working with so I could see that um well and then I guess I'd actually uh need to have some column names here as well but I could see well this is what um what I'm dealing with so let me do a data frame here I could see that it has all these columns here that have the one hot and turning and then I have these columns here that are doing um polynomial features on other things so this is the data that I'm using to try to predict what the height is so even though I'm kind of starting with just these two columns after I do all this transformation I'm actually giving um I'm giving the linear regression here at the end of my pipeline quite a bit more information to work with and that's why we're able to do much better um better here so as the very last step um what I would like to do so I saw that um model 4 was clearly the best one right so that's the one I'll recommend now it's possible if I'm doing four different models one of them just kind of by lock does better and that's why I hung on to my um uh to my training data right I only used uh around my test data I only used training data for cross validation scoring so at the very very end I'm going to fit this data to these things just like that and then let me just run that and then I could use it to make predictions right let me just remember how to do this because it's useful I could use this to make predictions on my test data frame like that I can make all these predictions for what the waves could be or I could um as a convenience score how good those predictions are against my test the f of wave height and I see while now I'm explaining uh 8.4% of variance right so I did a little bit lucky there but still I can see this is clearly better than my other models were hello so when we started learning about machine learning the first kind of problem we talked about was regression and after we learned about regression we learned about some of the linear algebra underlying it and now we're going to be moving on to a second kind of machine learning problem which is classification and and so I just want to review the main categories of machine learning um there's three main categories some people will say four uh but there's reinforcement learning um which is about these multiple stage decisions we're not doing that in 320 we're really interested in supervised machine learning which is where we're trying to make some sort of prediction about uh maybe the future or about some other unknown and then there's unsupervised where there's no particular thing that we're trying to predict but we're looking for some sort of patterns or simplicity within the data um we've been learning about regression which is where we want to predict the quantity and so we're going to be learning about the other um most common kind of um supervised machine learning which is well how do we predict the category and that's called classification so just to review these the two differences um here I have a big data frame and all of these things here are features and among these features I have a mix of both um uh quantities and categories and that's not really relevant looking at my features to figure out what kind of problem this is but I want to think about what kind of problem I'm dealing with I'll look at this label what am I trying to predict is that quantitative or categorical in this case it's quantitative so this is a regression problem and so what will we do I might have some data up here where um we're both my features and my labels are known it's all fit a model to that and then I'll use that same model to predict uh perhaps where that label is unknown or I might pretend it's unknown so I can basically test the effectiveness of my model if that's some sort of test data set the classification problem looks very similar again I might have some mix of um of quantities or categories as my features um the main difference now is that I have a categorical um Y or label column otherwise I'm still going to be fitting my features to my label and then trying to do some sort of prediction on it so um as I mentioned right we have these four big categories and uh and sklearn has so many different algorithms for each of them or implementations for each of them and so the one we've learned so far is linear regression that's what we've been using for um regression so linear regression model is what we call a regressor um very confusingly what we're going to be learning now is called a logistic regression and it is not a regression it's actually a classifier right so the name says linear regression but it's a classifier so don't get confused right even though we're learning linear regression and logistic regression I am teaching you a regression a regression model and then also a classification model so I'm going to exit out of here and head over to my notebook to try to introduce this and let me see here here is my notebook I have some stuff important maybe I'll come back to that later um let me just jump down here for now I have this data frame which has some data from a very famous machine learning data set called the iris data set the idea of the iris data set is that you have all these measurements about iris flowers so for example what is the size of the petals what is the length or width of the saples which are I guess like the green leaves between the petals um there are different varieties of um of irises so I put the varieties here in this far right column this whole thing I'm looking at right now is a test data set and it only has 10 samples in it um which is tiny right normally we'd have much larger um test data set but I'm just trying to keep it small and simple um in this case I'm passing in this random state so that even though it's um somewhat random I mean every time I run it will be the same if I put a different number here I would get a different random and I'm using random kind of carefully uh order each time so this is just so I can reproduce it even though I want something basically random I want random but reproducible so I've done that train test but I'm just putting 10 here and so what I'd be doing is we're going to be um fitting different models to the training data and then uh just seeing how they act with this very small um test data set and um looking at this there are three features I'm going to be interested in we're going to look at the dimensions of the samples and then I have this constant column um remember that sometimes when you have these models you could have coefficients in a separate intercept or you could just have coefficients and then the last coefficient could be multiplying the one column and that's what I'm doing here I think I'll make the later examples a little bit simpler those are my features um I actually have multiple y columns here I'm just trying to see if I can predict different things over here can I predict what the pedal width is and I predict whether or not a particular variety um is a sitosa um there's actually three varieties um in general so can I just predict any variety you can see that whenever I have a sitosa here uh it's true for other cases it's false right it's not a sitosa so that's what I'm going to do I'm going to see if I can predict these three different columns based on these three different features I guess it's really like two features and so there's four things we're going to do we're going to do a regression on the pedal width and that's really just kind of a review we're going to do a binary classification on the sitosa column and so we're going to try to predict whether it's true or false binary means two and so that's why there's just two things here all it's either true or false I'm going to use that same model to not just tell me whether it's true or false I'm going to ask the model for some sort of probability that it's true and probability that it's false rather just say hey it's true I'd like to see something like oh there's a 95% chance it's true and then finally so binary means two multi-class means um I guess more than two and so we're going to do that over here right you can see I have three different categories here and things get a little bit more complicated in that situation okay so I'm going to head down here and I am going to first just start with the regression and so I'm going to say regression equals linear regression this is just a review from before and I have all these options in here and this fit intercept one is something I'm actually going to turn to be false so fit intercept equals false what this normally does if it's true is it would add this one's column for me effectively when I'm saying false because I'm doing it manually and that just is going to make my example cleaner later on so I have this thing and I want to fit it to some data so I'm going to say regression.fit and when I fit what do I do I have my x and my y and then after that I could do regression.predict maybe some other axes and then that would return a y so I have something like this maybe I'll say like y, y2 and x2 so in my particular example I have my training and test data and so what am I trying to predict right now I'm trying to predict the pedal wet so I'm going to copy this column name and I am going to when I'm doing my when I'm doing my fitting I'm going to pull that out of my training data so I'm going to put that in here and quotes that's my y and then my axe it will be it will be again my training data and then I have to have some columns here I guess a list of columns and so I actually already created that right here I have these axe columns those are the sepal length sepal width and then constant and so this will be my list that I put inside of here and so I'll basically get those three columns let me just so if anybody is having trouble visualizing I'm getting just those three columns out of my bigger one whereas when I'm doing this I'm going to get a panda series that contains that pedal width column so I'm trying to predict this based on these three things okay so I'm doing that and now I want to do some predictions down here and that's going to be the same right I'm going to put in axes basically the same format except now I'm going to use my test data and maybe rather than attach that a variable it looks like for now so these are my predictions for what should go in this a pedal width column and so if I wanted to I could maybe even add those into my training data frame my test data frame and I could say my prediction equals that and then I could look at my test data frame and the thing that it's complaining about is that I'm trying to add some values to a slice of another data set so when I did this here these are slices of the rows inside of my big data frame and so if it's confused when I'm trying to add columns to one of these that's not the original so it's actually an easy thing to fix I can just say like test equals test.toppy and then test will be completely detached from my data frame and I can add columns to it without complaint so let me run that and now I can see I have this prediction column over here and I can go through and see how these predictions are so I predict 1.3 it's actually 1.2 I predict 1.5 9 actually 1.4 and so I can see sometimes the predictions are good and sometimes they're well not so great anyway that's a regression let me let me go on and try this next piece I want to do a binary classification on this column right here and so the code is actually very similar to before I'm going to head down here I'm going to change a few things first off I want to have a logistic regression and remember it is a classifier despite the name so I can deal with a category like this and then for my y I'm going to have this satosa column is it a satosa variety or is it not and so I'm doing that I may also rename this just so we remember it's a classifier I'm going to call it CLS then down here I also need CLS and I do that and now I can see my predictions here I can see that this column is telling me what the flower actually is and this column over here is telling me what the model predicts it is and actually I guess we're doing quite well here is completely accurate which is great all right let me go back up here so we did the regression using linear regression we did the binary classification using logistic regression and then basically saying well do I have a true or false now what I'd actually like to do is know what is the probability of getting a true and so I can do that like this I can say this has an extra function it's very similar to the predict but it will be like this it will be predict, probability, A and I'm going to get an umpire array of all the probabilities and so what this means is the way I interpret this is that there's a 94% chance of false and a 5% chance of true and that's why it ultimately reduced that to a false 97.9% chance of false and only a 2% chance of true that's why I get another false let me look at this one this one had a 93% chance of being true which is why I have true true there and so I could even, if I wanted to I could try to pull out that last column I could say something like I want to have some sort of slice and I have like a row slice and then a column slice when I'm doing this I want all the rows and I want that second column and so if I wanted to I could say something like test test of probability equals that and then I could look at it again and then I can kind of see in each case well based on these dimensions what probability does a model think it has of being a satosa and so sometimes it's not quite sure right I could based on this I could identify the cases where the model was not very confident and then I could identify other cases where it was quite obvious what it was so I don't have to do a new model for that I just have to call predict probability a instead of predict and okay so let me head up here and we're going to do this last piece now how can I do a multi-class classification on variety and variety is a little bit trickier because I have three different categories there I guess it's going to be trickier when we actually get into the math behind it but it is not trickier in terms of it is not trickier in terms of actually running the code so if I actually copy this here and then I head down here I'm just going to call this multi so I can remember my three different models let me say multi and and then what else do I need to change I'm doing a different column this time and so I can do that and now it's going to tell me for my predictions I see my predictions are still false and true and that came from here and so this is returning true and false because all my new model is called multi and I'm using my old model I'm going to do that and now I can see while writing that I can see what was the predict for this one that I wanted to write here it actually made a mistake models make mistakes that's not surprising I guess that was the only mistake in this data set and then of course this probability column was from earlier so I'm just going to ignore that because I keep on writing this probability or this prediction column for each of my four examples okay so there we've seen a few different things we've seen a regression and then we've seen these two different kinds of classifications let's try to get into the let's try to get into the linear algebra behind each of these examples and so I'm going to head down here and remember that I had REG that was my model earlier and inside of here I have coefficient and I also have intercept and the intercept is zero because earlier what happened I passed in that intercept equals false to all of these if I had not done that then what would have happened then my coefficients I only have these two columns here or two numbers here corresponding to the weights on these two columns and then instead of this being basically my intercept that would have gone to the separate variable down here instead of being here so basically these are my weights on my real columns and then this is the intercept or the weight on my one's column okay so I have that thing and what I'd like to do is reshape it so that it can be I want it to be however many rows necessary and then one column so it'll be vertical like that the other thing I want to do is I want to get my X data which will be my will be my test data frame and then it'll be those X columns and maybe I'll say dot values and so I'll look at that so I'm pulling those first three columns out of here okay so I have three columns here and then the column over here is basically three entries in it and so if I wanted to I could take the dot product of this with these down here and that's exactly how linear regression does predictions so if I come down here so I can say define regression predict and if I have some X values here what I could do is I could return X dot basically these things right so I could say um maybe I'll call this like vector one or I'll call it probably efficient you know what I'm just going to actually why not just put it here directly I don't need a separate variable for that and so I'm going to look at this regression predict and I'm going to do it on my X data and I get these predictions 1.32 1.59 let me actually draw up here earlier and I see that those are exactly the predictions that my linear regression made earlier 1.32, 1.59 right so this regular regression dot predict like this all it's doing is this right here it's doing this math right like this ok let's try to see what the logistic regression is doing so it's actually going to be very similar so remember before what was I doing I was saying CLS my classifier dot predict and um I can just do that these were the values I was getting out of my classifier what math is this doing it's actually almost identical to this let me copy this in fact I'm just going to tweak it very slightly I may have my CLS predict and I may say CLS predict my X data and um and the difference right I have all these numbers now and actually sorry I have coefficients from my other one so my CLS coefficients I should look at those as well before I jump into this I may have different coefficients why are all the numbers the same that doesn't make sense and I get all these numbers and remember our goal is to predict false false or true or things like that and so the way it works is that we're getting a score for each entry and if the score is greater than zero we predict true otherwise we predict false and so the shape is a little bit different here maybe I can just reshape so that's more obvious but otherwise that's what it's doing all these numbers up here are the same same down here and I think maybe this is why they maybe they even call this logistic regression even though it's a classifier the math is basically identical to what we have for a linear regression right at the heart of it we're just doing a dot product the only difference between the linear regression that we did before and the logistic regression working now is we're just checking if some number is greater than zero okay so let's do this next piece so the next piece was I'm going back through my examples before after we did predict which was trying to say are we predicting true or false we wanted to get this probability that's a setosa and so how can we get a probability out of this if I head back down here you actually see that before I added this greater than I had a numeric score so I'm going to go back to that and I'm going to say I want to predict probability A now I'm just going to go back to this and predict my probability and I see I have all these scores and of course these are not probabilities because a probability would be between zero and one but it turns out there's a very simple function that can turn it into probability and that function is called the sigmoid function and I had it at the beginning of the notebook but I haven't talked about it yet I'm just going to head up briefly and we're going to talk about the sigmoid function and how we're going to use it we don't have to go into a lot of the details on the math I don't care if you remember that I certainly don't remember it the important thing for it is that the X value that's growing in can be as large or as small as we want but if I get very very negative numbers I effectively approach zero and if I get very large numbers I effectively approach one so the nice thing is that I could take any sort of numeric score and it's going to give me back a result between zero and one so basically I can take some other sort of score and turn it into something that at least looks like a probability and so what I'm going to do is I'm going to call this sigmoid function down here and so instead of having all these numbers like negative 2.76 or 2.57 I'm just going to take the sigmoid of all of those things and then they will be these numbers all between zero and one and it turns out that that's how we were would actually be doing the probabilities so just like before I was trying to say well, predict the category on X the same way down here if I maybe I'll do this right here above I'm going to say predict the probability of A let me just look at what's happening here these numbers right here which is the probability of it being true is this column right here the probability of it being true the way I've written this code I'm not computing the probability of it being false because well that's boring I mean they add up to one so why would I do both of them but you can see these numbers here are exactly identical so getting the probability is very simple I'm still just doing that core dot product and then I'm just applying the sigmoid to every number inside of the result so we've seen for all these cases so far this dot product is just extremely important but I'm taking the dot product of a matrix with a vertical vector let me go up and talk about the fourth model we did with a multi-class model the multi-class model was right here and I was trying to predict variety and variety could be any one of these three things and what is going to turn out is that my coefficients for variety are going to be a lot more complicated so if I look at this is my binary one if I look at the coefficients for my binary one I just have these three numbers so if I look at my coefficients for my multi-class I actually have not a vector but a whole matrix with all of these numbers in it and so let me use these now and what we're going to do is actually just like the way they set it up we're going to have to transpose them we're going to do that and I'm going to have this here it's going to be very similar again but instead of multiplying x by just a vertical vector I am going to multiply it by this whole thing right here we multiply it by that whole thing and then I get this result and so I'm going to just draw this quickly I'm sorry this is multi-predict now multi-predict and I'm going to call that multi-predict of x and I'm actually not doing a sigmoid anymore at this point apologies and so how do I interpret this basically every one of these columns corresponds to one of the three varieties if I had had more varieties I would have had more columns up here but I would have still had three rows in this coefficient matrix so you might imagine I don't know which one is which but you might imagine that this is the Satosa variety and it turns out when I take my big matrix of data and I multiply it by all of these columns right here what it really does is it draws column by column it takes this column times my data matrix and it uses that to produce this output column so these might be the scores for how much it looks like a Satosa and maybe let's say these are the coefficients for a VersaColor then these would be the scores for a VersaColor if these were the coefficients for a Virginica then these might be the scores for the Virginica and so what I can do is that each one of these rows corresponds to a row of my original data which means this row corresponds to one particular flower one particular iris I have a score how much is it like a Satosa, how much is it like a VersaColor how much is it like a Virginica and so what I can do is I can try to take these and I can see which score is the highest and in this case it's this middle one so whatever whatever type of iris corresponds to that middle column is what I want to predict and so I can check that I could say if I go back to this I could say I think it's classes I could see okay well I guess well this is the Satosa and so that first one would be a VersaColor and so one of the ways I could actually do this is it turns out that there is an arg argmin let me just try to do this here let me try to get it right I am getting it right and so what I guess I want argmax what this is going to do is it's going to give me which index is basically giving me the largest value but I have to specify what axis and zero is down and one is across so I'm going to say one and what is this telling me I'm going to reshape it so it's a little bit more obvious I'm going to say negative one one so what this means is that at index one I have the biggest number in that first place the second position the biggest number is here is that true? yeah that's true this number is bigger than those other ones let me check the fourth one that means the zeroth position is bigger so this is bigger than these so it's absolutely true and so what's cool about if I have a numpy array like this I can put that into another numpy I can take these classes and I can put this array of numbers into here and I'm going to get basically what are all these categories and so I'm going to take all of this and and I'm going to put it back in here and I can see that then I'm going to get the predictions for all of my for all of my flowers I can see for each of them what am I predicting it is and this will correspond with my predictions earlier so again that product is at the heart of it but now since I have different possible classes I have to have coefficients to get a score for each class and so that's why we have to multiply the data by a full matrix instead of just a vector and that's the first example we've seen like that a real practical use case in this course in this video I want to talk about how we can visualize the decisions or predictions that are made by a classifier when we are doing regressions we often visualize this by drawing a fitline and the equivalent here is that we'll often at least when we have two features is we'll draw those two features on an x and y axis and then we'll separate the area into two different regions one where we predict true and other area where we predict false and so the function that we'll be using in map.lib is called contourF and it can do that kind of plot so we want to evaluate CLS and the idea is that we want to put in a bunch of different values for for both sepal length which I'm going to put on the y axis and sepal width which I'll put on the x axis and so the way to get every combination of these maybe most easily is with a numpy mesh grid and the reason I'm using a mesh grid is that it creates some arrays with different combinations and exactly the form that I'm going to need later so it creates arrays and form needed for contourF later and so let me actually leave this comment here like that and what is it trying to return well it's trying to return two arrays it's trying to return one which is one of my variables and the other will be the other variable so maybe I'll say I'll put my sepal a width first and then my sepal length just like that and then what I have to do here is I have to have a range right so I'll have like range one and range two and really what the mesh grid is going to do is it's going to give me every combination of these two ranges so here I'll say np a range maybe I'll go from zero to zero to ten and maybe do a zero point one step and then the same thing over here as well okay so I have those two and let me just take a look at what these things look like I see that both of these are a two by two matrix that are showing me every combination so let me look at the other one as well so these are exactly the same shape and the values in it are just giving you the coordinates right so the first one is giving me the x coordinate basically the sepal width and the other one is going to give me the y coordinate or the sepal length and so if I wanted to I could then call plt.con tour f and this is taking three things it has to have my matrix of my x coordinates and it has to have my matrix of my y coordinates and then it has to have something that says that gives the color and that can be some sort of expression so I could say if I wanted to sepal w and this will show me some stripes from left to right or I could do sepal l and that will show me vertical or I could have some sort of mathematical expression like this and at each point it's showing me what would happen if I multiply the value in one of these by the value in the other so I can get these nice contour maps now what I'd really like to do is to just have two levels two numbers let me just show you what this is now lots of different numbers I'd like to just have two numbers here basically one and a zero that correspond to predictions so my x axis is going to be the sepal width and my y axis will be the sepal length so I have to get all of this data into a format where I can do some predictions so let me leave this here for now and I may come back to this this is my goal where I'm working towards what I would like to do is put these things in a data frame so I'm going to say this will be my contour data frame data frame and my data frame has to have all of these things up here because I want to do predictions on it and these are my x columns and so I'm going to have something for that I'm going to have something for that and then I'll do my constant first that's easy, that's just one these values I can pull from these and so I can put this here and then I could put up here I could put sap w if I wanted to now this is not going to quite work because these are those basically square matrices and down here I'm trying to put all of this in a data frame so I can do my predictions it has to just be a simple column so I have to flatten these so it's just kind of one dimensional and I can do that and then I can look at my CDF if I like and I can see what is going on here is I really have every combination of length and width and then I have my constant column so this isn't a great format for me to then do some predictions because I can just say it has everything that I need for my predictor I can say cls.predict just like that and then I get all of these values right here and if I wanted to I could add those to that data frame as well I can say CDF I could say predicted, maybe I'll just say prediction and I remember that was the pedal width is what I'm trying to predict I can say, or I'm sorry it was the category, is it setosa or not that's what I'm trying to do so I have that and now I'd like to be going down and doing my time to work plot down here so I can have this and let me think a little bit carefully here so if I have my set W let me look at the shape of these that's a 100 by 100 because that's what my range was I was drawing over 100 numbers this is 100 by 100 matrix this is 100 by 100 matrix this is also going to have to be 100 by 100 matrix right now it's just a small column so just like I have to convert these matrices two columns with every shape here now I have to go the opposite direction I have to take values and then reshape this to be matching the format of these right so I can actually just use these and say well my prediction should be whatever shape my X values are and settle line up nicely and so I can do that and now I can see well the two sides of this are going to correspond to a prediction of it being either a setosa or not a setosa let me plot with a scatter plot on top of here um I have my data frame here of all my original flowers I'm going to plot all of them ideally I would just plot only the training data on top of it so I can actually have a better sense of what errors are made but I only have like 10 rows of my my testing data and so I'm just going to plot the whole thing so I'm going to say a dot plot dot scatter and actually what I'd like to do is I'd like to separate this out so I may have something like setosa dot plot dot scatter and then I may have something like other dot plot dot scatter and then those things I can get just with some filtering or somebody say setosa equals data frame where data frame of variety equals setosa and then my other ones will be basically where it's not that so I'll say where it is not setosa okay so what am I going to do down here well I may say my x value is going to be um well I was putting sepal width there so I better put that sepal width on my x axis and then my y axis should be sepal length okay so I'm going to do that and I'm going to do the same thing down here for a moment and I see okay well I have my decision boundaries and then I have my two separate plots down here I'd like these to all be on the same plot and normally the way that we've done things like that is we would say aax equals this and we pass that down below let me just show you quickly what the type of aax would be if I were to do that it's this thing here this um quad contour set and we can not plot on top of that so if I wanted to reuse the same um axis some plot area what I can actually do is I can say matplotlib dot hit current axis and that will give me an axis some plot that I can pass in elsewhere right so I could say up here I could say um uh maybe I'll go on to the next one I could say aax equals that and same thing here um so then they'll all go to the same area so let me do that and so I have all of these points and um and so then what I like to do now is um well it's kind of strange that it's not overlapping that boundary um did I get my axes mixed mixed up I did so here a sepal width sepal length okay great so let me let me just switch this back so this should have been sepal length and this should have been sepal width and now I can actually see that makes a lot more sense I can just try to separate the ones that are from the ones that are not and so to actually make this work now I should have the color be different in some way so I'm going to have the setosa's be uh red and then maybe I'll have the other ones be um I don't know maybe it can be gray right so just like that and so now I can start to see um what mistakes will be made I can see that there's one setosa that is not going to be recognized as a setosa because it's on the wrong side of this boundary um I should probably also have some um labels here so I'm going to say like label equals setosa and then down here maybe I'll say uh label equals equals other just like that and so I can see what's happening here now and so there's a couple steps to all of this right um maybe I can delete these extra things up here so I can have a minimal example I had to create a mesh grid and that basically for every point it one grid had the x values and the other one had the y values um I had to reshape those to try to convert them to data frame uh columns and at that point I really had every combination of these in some row once I did that I could add predictions for basically a recombination and then if I converted um um my predictions back to the mesh grid formula like these two have then I could do my contour and that's how I could create this map uh and then on top of that I could plot my scatter points and see what's happening one last thing I want to do here and that is I want to try to do some polynomial um uh vets so just like we can use polynomial features for a regular regression I can use those for classification as well and so let me import some stuff I may say from sklearn uh dot pipeline import pipeline and I may say from sklearn dot preprocessing I may say import polynomial features okay and so before what did I have I had um I just had a logistic regression like this with um with what was it it was um I was just going to search up in my page I think it was this fit intercept thing that I had off for all of them excuse me so I had that this was my model before and that's going to be part of a pipeline right so now I'm going to say pipe equals pipeline and the pipeline is going to be this list of stages just like this and so that will be the first stage well I'm sorry that will be the last stage and then before that I want to have my polynomial features just like that and um then the other trick is is that each stage of my pipeline has to be a tuple and the reason it has to be a tuple is because I have to give it a name right so I guess I'll just call this poly and I'll call this one lr so I have that and it's straight and so just like before I could have my um before I had something like this fit and then I had uh train of x columns and then train of what was my y column I guess it was um uh it was going to be my pedal width I think so let me let me just copy that it was like this but it was pedal width that's what I was doing before and I could just replace my uh logistic regression with this pipe so let me try that and I get some sort of error here unknown label type continuous oh I'm sorry I'm trying to predict the whether or not it's a sitosa right it's complaining because it's continuous so it's saying hey you're trying to do a classification on a quantity which we don't do classifications on we do regressions on quantities so what I'm going to say here is is it a sitosa that's what I care about and then I train it and that's all great now if I want to I can come back here and I can repeat these all these steps right so I could um if I wanted to maybe I'm just going to move all of this up here and then when I'm plotting this when I want to look at those decision boundaries instead of using my simple classifier before I could use my pipeline classifier and then what's going to happen well that boundary line between them is just slightly more curved it doesn't quite help that red point is still on the wrong side but um you can see I can do different shapes depending on the complexity of my model and what I have in the pipeline beforehand well on this video I'll be talking a little bit about how we can score our models um on our training data and um scoring them is very involved learning some new metrics and terminology uh it turns out there's different kinds of errors and um and if sometimes we'll look here more about one kind of error than another uh we might want to use different metrics to deal with that case um so for this I have some really simple dummy data uh basically I have this data frame um that has an x column and the x column is a number and then the y column is a is a boolean and it's true whenever x is positive and false when it's negative okay so not much data there and um and just for simplicity I'm trying to break it into the first half and the second half I wouldn't normally do this right because maybe the data is not shuffled and maybe I'm kind of getting very different data in the two halves so this is just trying for an example there I have it I have my training data and my test data down here and um and if I want to try to figure out what is the relationship between y and x and then um measure uh the model's understanding of that relationship um I'm going to use some sort of scoring function right so I'm going to do this whole logistic regression um however this is not really a regression it's actually a classifier right because I have this categorical data that I'm trying to predict and so I'm going to train one of those and then I want to score it so the first step right is I'm going to say l r equals logistic regression right and um and then I'm going to say l r dot fit and when I'm fitting let me actually just try to run this uh once I run I can actually set shift tab and into the hint I'm going to have to give it the x data and the y data and so I'm going to give it both those things from the training data first and so I'm going to say training data and um and then the columns I want are just x and then the thing I'm trying to predict is just the y value just like that and then after I do that I can basically run a command that's very similar and that command is score I can evaluate how well it does um the test data and I get 0.75 um so what is this score function doing? it turns out it's a shortcut and I'm just going to show you a little bit in the documentation what it's doing if I head over here I see that the score function for the logistic regression is giving me the mean accuracy and if you look at other estimators um they might be doing other kinds of uh using other metrics for the scoring okay so mean accuracy um so that's the default here uh but it turns out there's lots of different metrics we could use and so if I go to the metrics page for um scikit learn I can see that hey there's a whole bunch of um metrics here related to classification uh clustering which we haven't talked about yet and then um a whole bunch related to regression as well and so the one I'm using right now the default is that accuracy score right you saw that how it's saying here that we're um just getting the accuracy and accuracy is very simple so what percentage of the times um did we get it right and so I certainly could have instead of using that score function I could have used this one myself and I think that's a good thing to do right now because once you understand how to use this manually uh without the shortcut then we'll understand how to use these other functions as well okay so I can see this is the metrics um sub module and so when I head back here I'm going to do that so I'm going to say uh and this was that page I was just on I am going to say from uh from sklearn dot metrics import accuracy score okay and um and let me just run that and when I call accuracy score I hit shift tab here um basically what I'm doing is I'm saying well what are the true values and what did my model predict okay and and so there's different ways I could do that I mean I could say well the true value